There are three sections in the remaining of this tutorial:
1. Introduction.
2. Unpacking Custom Protection, Rebuilding
3. Cracking internal Protection, Fixing Registration System.
4. Conclusion

Ah yes.. another "bomb ass" tutorial.  This one will hopefully not be too long.  Although I say this now, soon I will dread the very thought of it. :)  Today's topic is Unpacking TMPGEnc DVD Author and cracking its internal protection system.  This is a very nice protection, which took me damn near 3 hours from start to finish.  The outer shell sorta resembles ASPack.  If it is then stupid me, For the purpose of this tutorial I will label it as Custom.  Anyways .. on with the show....

Now this sorta a ASPack clone.  But since the other apps from this same company don't appear this way.. rather a PUSHAD only... I will keep it as CUSTOM.  So our goal.. Get to the OEP of the real exe.  To do this we will use the stack for some help, instead of tracing by hand for a bit.  So execute the PUSHAD with F7 and then we will follow the ESP register in dump, and set a Hardware BP on Access WORD. 

Now follow ESP in dump.

Now select the first 2 bytes and then set a HW BP on ACCESS WORD.

Now we can hit F9 to break on our next Breakpoint.

Then hit F9 to break on the Next encryption layer.

Now this looks more like ASPack. :)

So hit F9 once more.

Now just execute the JNZ and follow through till RETN Using F7.

And we made it. :).

So you can safely dump this with OllyDump and then Attach its imports to it..

Then save it as whatever you want.  Now start up IMPREC and lets get our IAT.

Select our process and attach IMPREC to it.

Then enter in our OEP (005C3BA4 - 00400000 = 1C3BA4)

So in OEP box type in "1C3BA4" then click "IAT Autosearch"

Then click "Get Imports"

Then our Imports should all appear.

Now click the "Fix Dump" button.

And then save our Import Section to the exe we just dumped.

We now have a "Unprotected" executable, That just needs some things changed in it in order for it to run fully.  So tune into next section where we will Fix the CRC checks, and crack the registration system. :)

Bam.. you get this error.  I cant read it myself.. :(  but so you know.. this is a CRC check error.  We need to somehow get this exe to at least run to the regitration screen.  So we have a couple of options here.  One.. we can try to use the Call stack to see what kinda calls produce this error.  or two.. we can search in the Referenced text strings to see what's up... Well the second option is a tougher one here.. only cause this particular app is rather large (Delphi) and bloated.  We can try but you'll see a ton of useless shit here.  So rather we'll use the Call stack.  So with this error now showing (and it's being debugged through Olly) hit in this sequence ... F12 (to pause the process), then ALT+K (which will bring up the Call stack).  And if you didn't fall in the categories above.. Lets get with the program.. Shall we?  k... Start up Olly, Load the unpacked exe and hit F9 to run it, then hit F12 to pause, then ALT+K to bring up Call stack.

Well we can see our MessageBox being called here :)

Now we really don't want the actual MessageBox Call but rather the Call leading to it.

So right click the line labeled as "? dump_.00548E30"

Then we want to see our Call.  So select "Show Call".

And there's our Call we want.  So the idea is simple.  Somewhere inside this Call is our CRC calculation.  We just need to find it and then patch it.  So hit "Enter" on the Call to enter the destination.

So its quite obvious now isn't it?  The MessageBoxA being called is our CRC Nag Error.  That JNZ above is our ticket out of this mess.  So somehow the JNZ needs to be satisfied to JMP over this MessageBox.  So the next step?...... enter the Call above it and see how we can make EAX Return 1.  So highlight the Call at offset 00548E57 and hit the "Enter" button.

Hit "Enter"

Now this isn't much to look at this second.  We might want to run the exe and see how it responds to the CRC as it is.  But if you scroll down a bit you can see it.. :)

Obviously the strings were cut a bit to fit this screen.  But you get the idea here.  Somehow our check here must be a satisfactory one, or even leave here bad but satisfied after it (as long as we reach the JNZ with EAX=1).  So the easiest way to figure this out would be to put a Breakpoint at the PUSH EBP and let it run.

So we hit F9 to run it.  And we break on the PUSH EBP.

Then we continue to trace this out till we get an idea of what the program is doing to create this error, using F8 (Cause every Call here isn't important to us).

Once we get here (A little ways down) we see a important area to us.  That JMP will Jump over the whole entire CRC Check.  So we must make the JMP never happen.  Above it are a few Conditional Jump's.  One of these will need to be patched so we never Jump over this CRC Check.  So for instance this Jump:

Will be a problem for us.  It Jump's directly to the JMP.  SO by simply "Nulling" the Jump we can override it.  I am not sure if you already know this but there are 2 ways to NULL.

1. Which is well known.  is simply to NOP (90) the instruction.

2. Placing a 00 where the distance is in the Hex code. (eg. 7E09)  By simply making it 7E00 it does the same as NOP.  So the jump will happen , but will jump to the next instruction instead of 9 Bytes.  Make Sense?

So I chose the latter here.  So all we can do is right click the instruction and follow in dump.

Now we see the hex dump:

and then on the "09" hit the "Spacebar" key.

Then just type in "00"

Then clcik the "OK" button.

Then if we look at the instruction we see it jump's down to the next instruction.

Now that's done. :)  But that's only the beginning :( .

So now continue to trace with F8 till we reach another important area.  Which will seem like a while here.  You will reach the CRC check algo and pass it up.  Then reach a few more Conditional Jump's.  These are very important. :)

You get here.. We need this to jump.  so trace it through and make sure it does.  If not we'll need to make it jump.  But when we trace it we see it DOES jump.  So no need to patch this also.  So we keep moving on.

Then we reach another area of interest.

Obviously... it looks real similar to above.  3 Calls in a row.  And it would Jump over the rest of this operation, thus making EAX return 0.  So this MUST jump.  there is probably 100 thousand solutions for this.  the way I came up with was to MOV 3F instead of comparing, therefore not messing with the CRC Check's algo.  So if you look below you see what i did.

I made it MOV the value to the pointer like SHOULD already be there.  Plus I made it JUMP.. just in case.  Yes I know.. Some bytes below were used as extra.  But who cares?  They would get Jumped over anyway....Right?

So this will get us past this calculation and hopefully we can move on the rest, where.. we can hopefully finish up this CRC Check.  So continue to trace it out with F8 till we reach this spot:

Yes I know we skipped a few "Important Looking things".  But believe me they hold no great value here.  All we needed was the above patches plus this next one coming up.

We see it is gonna move EBP-1 to AL.  Well if you trace it you see it moves 0.. This is not what we want here.  We want 1.  So simply doing this:

We will satisfy the JNZ after we execute the RETN.

So after you patch it... Trace it to the JNZ.

And we beat this CRC check. :).  SO go ahead and run it..

And yes.. we defeated this CRC check. :)

Good job.  Now all that's left is to patch the registration. So before we forget go ahead and save the changes we just made so our exe wont need patching anymore to run right. I will give you 5 minutes (go smoke.... whatever)

(5 MINUTE BREAK)

Okay we ready to crack the registration?  Lets get to it....

Load up our newly patched exe in Olly and then lets think real hard about how are we gonna do this?  Well we can try searching the string references for a certain string.. maybe a hint to it?

Okay so we search and search.. we really don't find anything good.  The only thing we can use is the Registry.  If you noticed when searching through the strings.. you saw a RegistrationCode string. Right?  but only thing is.. it says its for v1.0.. we are on v1.6.  Well I am pretty sure the string should still remain the same through the versions.. right?  Well why not create the string our selves and then restart it to see how it reacts to our Serial.. Then maybe we can patch it to accept it, or just think its registered.  So now you need to create a string like this:

REGEDIT4

[HKEY_LOCAL_MACHINE\SOFTWARE\Pegasys Inc.\TMPGEnc DVD Author 1.6]
"App"="D:\\Program Files\\Pegasys Inc\\TMPGEnc DVD Author 1.6\\TMPGEncDVDAuthor16.exe"
"RegistrationCode"="TD169988998899889988"

I chose the 998899's cause that's my standard code I use.  Remember it had the format like this:

Now we restart it after we create the key and then we need to break as it reads from the key we made.. So using the APIBreak plugin (thanks to Nonameo for this one) we seta  break on all the registry API's.

Then click "OK" then run it till we reach this area.  (cause most of it is garbage until we reach here):

So now hit ALT+F9 to return to the Caller.

Then it's safe to remove the Registry BP's we made  (only so we don't keep breaking on them while tracing and get confused).

So use the plugin again and click the "Erase all API breaks" then we can continue.

So obviously this area is not our intended patching area.  We need to get out of this spot (only cause this is the generic registry read area) any patches here would patch every registry read.  So hit CTRL+F9 to land on RETN, then hit F7 to get to next operation.

then F7 to get out of here.

We land here, but need to get out of here also.  So F7 through RETN and get here:

So we still are in to deep here.  Lets get out of here also by usinf CTRL+F9 a few times through each RETN in this operation:

Then through Next RETN.

Then again CTRL+F9 through next RETN.

And finally at the end of this.  Hit F7 to Execute RETN.

So we made it out of that operation. :)

Now we can see that call was our Call to our registration key.  So we can place a BP above it so if we get lost we can restart it and then start from here again.  But the Idea here is simple.  That "Registration" popup is called somewhere.  The idea is to find exactly where and reverse the operations around it, so it thinks it's registered.  So keep tracing with F8 till the nag comes up.

And this call is our bastard here :)  So what we need to understand a bit here.. is why it gets here.  And more importantly.. can we get around it somehow?

Well some good news.. :)  The JE above.... that is our target.  If it doesnt jump.. we jump over the registration system.  So we must see what sets EAX to 0 after we return from that Call.

Our Call is highlighted in blue.  So hit the "Enter" key on it to enter the function.

We are inside the Call here above.

Now we need to figure our why EAX will equal 0 upon executing the return.  So easiest way is to set a Bp on the PUSH EBP, and then run till we break here.

Then hit F9 to run and break here.

Now we can trace till we see something/anything interesting.

Well its pretty self explanatory here.  You you look closely... you see we Move 1 to the Lower EBX register (BL)  See it?  Well later on we move EBX to EAX.  which would give us 1 :)  so somehow we need to make the MOV BL, 1 instruction execute.  Don't look easy huh ...?  a lot of conditional jumps.  A rather large patch area here.  SO we will look at it closely:

Enter the call that's highlighted in blue here.

We see this call as maybe our ticket out?  Let's check where it takes us to. 

Well we move whatever byte that EBP-9 points to, to AL.  very simply we can always make it move 1.

by doing the following.

But now we have the JNZ above we need to patch.  So go back to it and then make it JMP instead.

Now this area is patched :).. lets go on.  get back to our original Caller. So hit the "-" key a few times till you return to where we were a second ago. 

Now this JE never will jump again. :)  Now we have a few more to take a look at.

Now here we see we need the same idea.  so hit the "Enter" key on this Call and enter it.

OUCH! this one looks tough :(.....

Well not really :).. same shit as before.  All we need to do it somehow get down here without any problems.  But it seems as though a lot of patching is involved here just to make EAX equal 1 here.  So instead we will just let the first jump we encounter just jump, and change the instruction at the bottom.  MOV EAX, EBX. 

Make it this:

And then the MOV EAX, EBX.... we simply do a MOV AL, 1

We do this:

Now we can proceed yet again... :)

So hit the "-" key again till we reach where we started from...

Now we face yet another Jump.. but this one is a bit nicer to the eyes. :)

If we trace it out and see where it goes to...

We are all good if we jump here.  So simply just make the JNZ -> JMP.

Now lets test our patches up the the return....

So trace them our with F8 till we reach the return.

We reach the Test before we hit the conditional jump here and check our EAX register....

And its obvious we wont jump here :) :) :).....  So we essentially make the app think its registered ..... So lets hit the F9 key and see what happens...

(image cut for size reasons)

WE DID IT!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Okay young grasshoppers... we have officially unpacked and cracked the CUSTOM Protection.. only a few people can say they can do....  BUT.......

YES .... BUT......

Try to encode something with it.....  What happens?

Yes it just quits on us.... Damn.. we almost had it... :(

Well not really.. When I said this was gonna be a long tutorial.. I meant it...  There is still one last check we need to fix.. And that is.... Another CRC Check......  But this check is identical to the first one we patched.. just the problem is.. there really isn't anything to go on here... Except for the strings.. You remember them????? SO lets be smart here and save our changes we just made.... So we don't need to repeat them again...  I will give you another 5 minute break.. Go smoke.. go take a crap.. whatever...

Okay.. you done?

Lets move on...  e learned earlier how to fix the CRC Check.  Well this is no exception... Do you remember the strings we saw (you don't need to remember them exactly).  But at least remember something.. this will help us..

So lets restart the Program in Olly and the once we hit the OEP or EP (only to not confuse) we search for all referenced text strings.. we do this by:

right clicking and the selecting - search for - all referenced text strings.

we don't need to look far :)

as soon as the text strings come up we see the string :)

So hit the "Enter" key on the first string.  You land here:

What we are looking for is the similarity from the previous check.

We found it.. but remember we had a Jump above here patched too.  Plus the MOV AL, 1 we did.  So go ahead and make the changes.

The jump is patched to NULL.

The Compare function is patched.. then the last thing....

And the last final thing.. thank god...

Now just save your changes and run it, try to encode now :)

Now after you load up a video file, hit the "Begin output" button...

Now we are done :) :)

Well I hoped you learned some sort of valuable lesson here.. Or else I failed...  I had originally planned to release this on Memorial Day.. but time beat me...

This tutorial is dedicated to the Men & Women who entered their eternal rest in Iraq.  To all the Men & Women who are still alive, but feel like they are dead mentally... And most importantly.. to their families.. May the deceased Rest In Peace.. Thank You for defending my Freedom.

Until next time I remain....

MaDMAn_H3rCuL3s...

1. You learned that just because it's a custom protection.. doesn't mean it can't be done.

2. It took me long hours to create this.. some constructive criticism would be nice :)

3. ARTeam is forever

[MAIN TEAM]
[Nilrem] [JDog45] [Shub - Nigurrath] [MaDMAn_H3rCuL3s] [Ferrari] [Kruger] [Teerayoot] [R@dier] [ThunderPwr][Eggi] [EJ12N] [Stickman 373] [Bone Enterprise]

[TSRH] [some 0day grps] [BriteDream] [Exetools] [CUG] [Ricardo] [SnD] [fly] [PEdiy forums] [MEPHiST0]